The cognition of comics: What "comics" can tell us about the mind

Drawing is a fundamental aspect of human communicative expression, and sequential images have been used to convey information since at least cave paintings. However, only until recently has increased research turned towards studying the cognition of visual narratives—especially those found in comics. The most examined question about visual narratives has asked: How do readers make meaning out of a sequence of images? Such an inquiry must balance the understanding of meaning within and across individual images, the generation of inferences from such content, and the engagement of such phenomena within the physical layouts of a page, possibly in multimodal interactions. Recent research has posited that sequential images use a “grammar” that organizes categorical roles into hierarchic constituents. However, rather than a grammar of “syntax” like in sentences, sequential images employ a “narrative grammar” at a discourse level (Cohn, 2013b) to order “panels”—the encapsulated image units of a visual narrative. Such structures are separate from, yet interface with, the rules organizing the composition and navigation of images arranged in a page’s layout (Cohn, 2013a). Ongoing research has demonstrated that, like the relationship between syntax and semantics, this narrative grammar is separate from meaning (Cohn, 2012; Cohn, Paczynski, Jackendoff, Holcomb, & Kuperberg, 2012), and is organized using a constituent structure (Cohn, 2012; Gernsbacher, 1985). In addition, manipulation of these structures yields the same brain responses as comparable paradigms from sentence processing (Cohn, 2012; Cohn et al., 2012). This initial research has raised several questions: 1) How do readers negotiate the interaction of text, images, and page layouts in their comprehension of both sequential images and individual panels? 2) What aspects of individual images might influence the comprehension of sequential images? 3) How does the comprehension of visual narratives overlap with the cognition of other domains? The presentations in this symposium will report on ongoing collaborative research addressing these questions, along with illustrating the potential that visual narratives hold for the cognitive sciences more generally.